Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: use toml for tox #17

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

henryiii
Copy link
Collaborator

@henryiii henryiii commented Nov 14, 2024

I am trying the (fairly) new native TOML support for tox here. I've not used it before, and got some help from the Tox discord (thanks to @gaborbernat!). Currently, you pretty much have to have 4.22+ to use it; tox-dev/tox#3402 would allow a workaround for (somewhat) older tox's.

Thoughts?


📚 Documentation preview 📚: https://dependency-groups--17.org.readthedocs.build/en/17/

Signed-off-by: Henry Schreiner <[email protected]>
Comment on lines +169 to +172
base = ["tool.tox.env.mypy"]

[tool.tox.env.mypy-py313]
base = ["env.mypy"]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, by the way, either seems to work, was testing. The config will show the full section name, tool.tox.env.mypy.

@sirosen
Copy link
Owner

sirosen commented Nov 15, 2024

I want to get a chance to play with this before approving/merging, which realistically means a couple of weeks wait time as I'm not really available right now. I hope that's okay.
I don't think we'd need any special compatibility (like a stub .ini file) for older tox versions. They'd simply be unsupported by the project, and I think that's fine.

Having worked through the conversion, do you see major benefits to the new TOML syntax other than having everything embedded in the pyproject file? i.e. If this were a conversion to tox.toml, what would the benefits of the new format be?

I ask because I've seen the new TOML support in tox docs and have been wondering about why I should (or shouldn't) convert. It looks like it's easier from the maintainer side, but ini support isn't going away (at least not soon?), so I'm not clear on the user benefits.
It does look like there are some environment inheritance options, and I guess we could parse and lint or even generate content with tomlkit? But those seem a little niche, so I'm wondering if I've missed some key improvements.

@henryiii
Copy link
Collaborator Author

No rush at all, or even expectation that it is merged, I mostly wanted to learn the new TOML configuration.

I'm personally fond of fewer top-level files, since they push the readme down on the page and make the repo appear more complex. It's also rather similar to uv and hatch, which can be configured in pyproject.toml too. But tox is a pretty good candidate for a top level file, so tox.toml is fine too, I think it's still a small improvement.

The main features so far are:

  • TOML is better than not-really-a-format .cfg/ini, things like lists are actually part of the language. Also, this is the last bit of ini/cfg left.
  • Less terse. A bit longer, but easier for a newcomer to read (I think), less magic.
  • Can be validated - I'm working on setting up a Schema for validate-pyproject and check-jsonschema (feat: adding a json schema command tox-dev/tox#3446)

(Personally, I'm still a fan of nox, and only use tox on a few projects - mostly ones that also have a tox maintainer too. ;) )

@sirosen
Copy link
Owner

sirosen commented Dec 15, 2024

It took more than a hot second for me to get the kind of free time to toy with this! But I've now gotten an hour or two to try it out.

I'm still on the fence about making this change, but I now have more informed thoughts, so I've definitely gotten value from this exploration.

  • The TOML config is less arcane to learn because it has easier to understand rules about things like {posargs}. Overall, I'm quite satisfied with it as a format for test runner config.
  • I agree that "few top-level configs" is typically a nicety, but I also know that one of my immediate heuristics when coming into a foreign project is "is there a tox.ini or noxfile.py?" I'm not sure I like losing that.
  • tox.ini data is pretty big (sheer number of lines) to try to squeeze into pyproject.toml, relative to other tool configs. Coverage is the nearest second, and it's much smaller. In this format, tool.tox is fully half of the file.
    • This is not inherently a problem, but it raises questions about dev ergonomics which I don't think other tool configs do. How big can pyproject.toml get before it's "uncomfortable"? When is it better to split this off into a separate file? (I don't have a clear answer here.)
    • I consider dependency-groups to be a simple package with simple tox config. So this tells me that for a more complex case, I really won't be able to do this. As a user with multiple tox projects, that means I'll need to know both TOML and INI formats.
  • Judging from the docs, we're losing a couple of features of the INI config. ( https://tox.wiki/en/stable/config.html#ini-only ) Of these, there are two which I note:
    • The generative env list and section names. I tried to refactor the TOML config (just to play with it) to declare mypy{,-py3.8,-py3.13} and realized this. It's not a major feature which ever project needs, but when you want it, it's quite handy.
    • Conditional settings based on factors! This is huge, and is, IMO, one of the most powerful features of tox. When testing packages with optional extras or soft-dependencies, it's invaluable to be able to write yaml: pyyaml in deps. However... we don't need it in this project yet, and it might be interesting to explore how to do this sort of work under the TOML config.

In principle, I'm in favor of trying new things, and being willing to revert if we find they don't work or we don't like them.1

But I can't deny that I'm not loving the new config, at least not yet. I have three ideas for paths that I would take for this config:

  1. try out this config
  2. try it out, but using tox.toml rather than pyproject.toml
  3. set this aside for now (close the PR)

(1) and (3) are sort of obvious. (2) is a middle-ground.

I'm going to take some time to try to decide which of these I like best, and I'm open to any more info or thoughts which might help make the choice.

Footnotes

  1. And being willing to try them again! I didn't have a good first experience with ruff but I won't rule out using it forever.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants